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This paper discusses a research agenda setting project conducted for an 
international non-governmental organization which aims to help create a 
regionally relevant, high-quality knozvledge base on key education issues of policy 
and practice. Specifically, we illustrate how our team adapted the Child Health 
and Nutrition Research Initiative (CHRNI) methodology for research priority 
setting to identify areas for future education research. The paper focuses on the 
methods used and lessons learned from adapting these methods for education 
research, as opposed to the results of the exercise itself. While the traditional 
CHNRI model is not directly applicable to education research, an adapted version 
of this methodology provides an evidence-based approach to support the decision 
making process of research funders. The CHNRI method does not provide definite 
answers on what research should be funded; rather it provides a solid platform to 
support informed discussion on ways research can be prioritized. 


Introduction 

The field of international development is founded upon the idea that highly specialized 
knowledge and technical skills have the potential to solve some of the most critical 
challenges facing our world (Easterly, 2013). In the twenty-first century, as the global 
paradigm has shifted away from a global economy based on material resources and 
other forms of physical capital, to a global economy based on knowledge as the new 
source of wealth (Stiglitz, 1999), the concept of evidence-based development or 
knowledge-based aid, has gained emphasis. Today, the ability to produce and use 
knowledge is seen as critical to a nation's comparative advantage (Government of 
Canada, 2001), and is a major focus in international development (World Bank, 2009). As 
a result, there is a growing demand for empirically based evidence and specialized 
technical advice to help developing countries and aid agencies to assess the impact of 
development activities. 

Research is valuable to both policy makers and practitioners in many fields. In 
education, empirical evidence plays a vital role in improving the quality of schooling 


1 The CHNRI exercise discussed in this article was undertaken through the Partnership for 
Advancing Human Development in Africa and Asia, an initiative of Aga Khan Foundation 
Canada with financial support provided by Global Affairs Canada 
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across the globe. It has the potential to improve both the efficiency and effectiveness of 
educational policy and practice by providing attested information to enhance the quality 
and cost-effectiveness of policies (Levin, 2010; Livingstone, 2005). In the case of 
international education development, limited resources combined with an increased 
culture of accountability has created a demand for evidence-based decision making at all 
levels. However, despite increasing calls for evidence-based development, there are 
many gaps in our empirical understanding of education development around the world. 
As a result, development organizations are orienting themselves towards supporting 
knowledge generation in order to fill these gaps (Piotrosky, 2014). 

The shift from funding programs to funding research to promote development can be 
difficult. Organizations struggle with how to generate a systematic and transparent 
processes for creating research agendas that identify not only what research is necessary, 
but also what research would have the most impact on the ground? How can 
organizations ensure that voices from the developing world are represented, and how 
do stakeholders ensure that the research funded is both methodologically sound and 
practically relevant? This paper discusses a research agenda setting project conducted 
for an international non-governmental organization which aims to help create a 
regionally relevant, high-quality knowledge base on key education issues of policy and 
practice. This knowledge base should, then, stimulate debate, shape policy, contribute to 
project learning and improve the effectiveness of education programs in a specific region 
of the developing world. Specifically, we illustrate how our team adapted the Child 
Health and Nutrition Research Initiative (CHRNI) methodology for research priority 
setting to identify priority areas for future education research. Rather than focusing on 
the results of the exercise itself, this paper discusses the CHNRI method and the 
concepts learned from adapting this method for education research. 

The Child Health and Nutrition Research Initiative (CHRNI) Research Priority 
Setting Method 

In 2005, the CHNRI, a project of the Global Forum for Health Research, identified the 
need for a, "practical and systematic tool that could assist priority setting in health 
research investments" (Rudan, 2012, p. 237). Over the next two years, a panel of experts 
(including researchers, program makers from low and middle income countries, and 
representatives of international organizations) funded by the World Bank, worked over 
a series of 12 consecutive meetings to develop a methodology appropriate for setting 
research priorities that was both scientifically rigorous and replicable (ibid). 

The resulting CHNRI research priority setting exercise is a systematic and transparent 
approach to research agenda setting which involves working with an array of experts in 
a certain field of research. In its original model, the CHNRI method limited the 
definition of research strictly to activities which resulted in the generation of new 
knowledge, but over the years this definition was expanded to include knowledge 
mobilization activities as well (Tomlinson, et al., 2007). The CHNRI exercise is a flexible 
method which can be used to prioritize research investments on a local, national, 
regional or global scale (Rudan, et al., 2008). This exercise can be implemented by a 
small group of individuals responsible for representing the vision and interest of 
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organizations which invest in research. Thus, this methodology is appropriate for 
organizations making sizable investments in research such as large INGO's, multilateral 
organizations, national governments, and other research funding organizations (ibid). 

While the actual steps may differ depending on the context, the CHNRI framework 
identifies five key components for setting research priorities which guide each iteration 
of this method. These include (see Tomlinson, et al. 2007; 2009): 

1. Gathering Technical Experts and defining the context (i.e. geographic 
scale, time period, and target population); 

2. Systematically listing research options by domain of research; 

3. Scoring all listed research options by a pre-determined set of criteria; 

4. Addressing stakeholders values (this could include weighting certain 
criteria in order to account for a particular item which is highly valued by 
the funding agency); and 

5. Program budgeting and, marginal analysis and advocacy. 

A key component of the CHNRI exercise is developing criteria for rating and 
prioritizing the list of research options produced. While these can be adapted to suit the 
needs and interests of the funding organization, CHNRI's standard criteria include the 
following five dimensions (from Rudan, 2012, p. 238): 

1. the likelihood that the proposed research question is answerable; 

2. the likelihood that the proposed research would result in an effective 
health intervention; 

3. the likelihood that this resulting intervention would actually be 
deliverable, affordable and sustainable; 

4. the maximum potential that the intervention would reduce the disease 
burden; and 

5. the likelihood that this intervention would reduce inequality. 

These criteria not only help to identify research priorities, but also help investors 
identify the potential benefits and risks of an array of research options (Wanzy, et al., 
2013). While the sponsor of this project chose not to, the funding agency can choose to 
weight one or more of the criteria in the analysis stage in order to address their specific 
values and interests (Tomlinson, et al., 2008). 

Analysis of the CHNRI survey responses includes computing the Research Priority 
Score (RPS), average combined rating for all respondents across criteria for each research 
question. The Average Expert Agreement score (AEA) is also computed which shows 
the degree of agreement and variability across experts on the ratings for each question. 
The RPS is based on a possible score of 100 points. The closer to 100, the higher the score 
and priority given to that question across all respondents while a score closer to 1.00 in 
the AEA suggest greater consensus. Both scores are used to generate a ranking of 
research priorities based on the level of agreement among experts. 

Overall RPS is calculated as the mean of score for each criteria per each question: 

[(Criteria 1 score)+(Criteria 2 score)+(Criteria 3 score)+(Criteria 4 score)]! 4 
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AEA is calculated for each research question following Dean and colleagues (2013) 
guidelines. They suggest calculating AEA as the "average proportion of scorers that 
gave the most common answer while scoring that question" (p.5), according to the 
formula: 


4 

< 7=1 


N(scorers who provided the most frequent response ) 
N (scorers) 


Note: Where q is 1 of the 4 criteria questions that experts used to evaluate 
competing research questions 


The CHNRI exercise relies on the collective opinions of a group of experts, as opposed to 
one expert opinion. The highly structured format of the CHNRI methodology and 
independent scoring ensures that no single individual is able to dominate the process, 
which limits the possibility of personal bias distorting the outcome (Tomlinson, et al., 
2007). Instead, the process identifies the points of greatest agreement, and as such, the 
CHNRI method is viewed as, "a widely applicable tool for consensus development in 
any area of research" (Rudin, 2011, p. 239). While the results of a CHNRI exercise do not 
determine where resources for research will be directed, the findings provide useful 
information that reflects and synthesizes the views of a broad range of knowledgeable 
experts in a designated field. 


Applying the CHNRI Method to International Education Development Research 

Originally used to identify research priorities related to child health and nutrition, the 
CHNRI method has been adapted to support research agenda setting for an array of 
health related issues such as global mental health research (Tomlinson, et al., 2009), 
reducing maternal and child mortality in low and middle income countries (Dean, et al., 
2013), Childhood Diarrhoeal Disease (Wanzy, et al., 2013), and determining research 
priorities on the issue of Family Planning (Campbell, Higginbotham, & Polonsky, 2014). 
However, we found no evidence that this methodology has been applied outside of the 
health sector. To our knowledge, this paper deals with the first time this process has 
been used to identify and prioritize research needs in the education sector. 

The CHNRI exercise has been traditionally used for specific interventions in the context 
of health research and is a model which cannot be entirely transferred to education. 
Thus, we had to adapt the CHNRI model to suit the needs of the sponsor organization. 
In turn, as detailed in the paragraphs below, the revised model is more appropriate for 
the complex and multi-dimensional nature of education research in the developing 
world. 


The Need for a CHNRI in Education 

In response to Education for All and Millennium Development Goals, many national 
governments and non-governmental actors in developing countries gave primacy to 
policies and interventions designed to increase access to primary education to all 
children. However, the experiences of our sponsor organization suggest that 
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improvements in access do not necessarily produce improvement in quality. Tangible 
improvements in quality contribute to better enrollment and retention, and 
strengthening the quality of education in schools requires a comprehensive whole school 
approach to school improvement that addresses these dimensions in a coordinated and 
coherent way. The success of school improvement inputs, such as teacher and head 
teacher training, should not be limited to evidence of participation satisfaction and 
implementation of new programs and practices. Any measurement of success must take 
into account evidence of student outcomes. Another key lesson is that the sustainability 
of school improvement structures and processes, and scaling up beyond the schools and 
education personnel initially targeted do not just happen. Sustainable education 
development requires strategic partnerships with local education system authorities and 
institutions, while at the same time working within, influencing and leveraging national 
education policy and resource realities. 

Many of these issues and lessons learned by our sponsor organization are aligned with 
findings from international and North American research on school improvement 
(Hargreaves and Shirley 2012). However, international research on school improvement 
increasingly emphasizes the collection of data and employment of experimental research 
and intervention designs to enable more targeted solutions to specific programs. In 
order to address this issue, the sponsor organization planned to invest in regionally 
relevant, high-quality research on key education issues of policy and practice which 
could contribute to improving the effectiveness of education programs in a specific 
region of the developing world. 

The sponsor organization had previously participated in a CHNRI exercise to support 
their work in the health sector, and recognized the potential this methodology has for 
research priority setting in other sectors. Our team at the Ontario Institute for Studies in 
Education (OISE) was brought on to support this project. 

The OISE team used the following approach to gathering and synthesizing information 
about gaps and opportunities for research priorities in support of the sponsor 
organization's objectives. This section is structured according to the five key 
components of the CHNRI method. Here we highlight some of the key ways we adapted 
this methodology to address the needs of our sponsor and the context of education 
research in the developing world. On this note, in the traditional CHNRI model 
'Addressing Stakeholder Needs' is listed as step four out of five. However, in our model 
we collaborated with the sponsor organization throughout the duration of the project to 
ensure that the project was addressing their needs. Therefore, in this paper we have 
included stakeholder needs under each of the other aspects of the CHNRI model as 
opposed to listing it under a separate heading. 

Gathering Technical Experts and defining the context 

In order to define the context of this project, our team at OISE completed a review of the 
sponsor organization's internal documents related to school improvement and to 
education in the specific region of focus for this project. During this phase, we identified 
six broad areas commonly addressed in the funding agencies school improvement 
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initiatives in the developing world. This included: a) teaching and learning, b) teacher 
development (pre- and in-service), c) school management and leadership, d) parent and 
community involvement; e) system-level factors, i.e., policies, human and resource 
capacity; and f) early childhood development and education. These six themes 
combined with the sponsor organizations focus on a specific geographical region of the 
developing world defined the context of this CHNRI exercise and were therefore used to 
frame the context for the rest of the project. 

In consultation with the project sponsor, our team also identified 84 education experts 
knowledgeable about the region and subject, internal and external to the sponsor, 
including a mix of local and non-local experts, and both academic and professionals who 
were later invited to participate in the project. 

In order to ensure our frame met our sponsor organization's needs, we hosted a full day 
meeting with a group of stakeholders from the sponsor organization as well as a select 
group of education experts early on in the process. Based on a discussion paper 
provided by the Principal Investigators, we discussed the six broad themes identified 
above, as well as the types of research questions and methodologies which had the 
potential to have a catalytic impact in the field. Insights gleaned from this discussion 
were taken into consideration during the development of our list of research options 
outlined below. 

Systematically listing research options by domain of research 

In a traditional CHNRI, the small group managing the project would be responsible for 
creating the initial list of research options. However, for our project, we utilized the 
approach of Dean, et al. (2013), who included two iterations of the CHNRI in their 
exercise. Based on this adaptation, we developed a short open-ended survey to solicit 
input about primary / pre-primary education research needs and priorities for a group of 
countries within a particular region of the world. All 84 education experts were invited 
to respond to this survey either online or by telephone. In total, we received 22 
responses (26%). This initial survey data was then combined with sponsor organization 
document summary data to generate a comprehensive list of 267 potential research 
options. 

In the health sector, it is common for CHNRI participants to rank hundreds of research 
options in a CHNRI exercise (e.g. Wanzy, et al., (2013) ranked 466 research options); we, 
however, did not feel that education experts would participate in our CHNRI if it 
required them to rank the 267 research options that we identified through phase one of 
this project. Therefore, for the second phase, we clustered and reduced the list of 
research options into 89 research questions categorized under key themes. 

Scoring all listed research options by a pre-determined set of criteria 

Based on a review of prior CHNRI reports and in consultation with the funding agency 
we developed a set of four criteria which were used for evaluating the research 
questions. These included: 
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1 . SYSTEM IMPACT: Research on this question will be catalytic / 
instrumental in improving system-level policies and interventions that 
support teaching and learning across diverse contexts in the developing 
world. 

2. SCHOOL IMPACT: Research on this question can lead to substantial 
improvements in school level practices that support children's learning and 
development across diverse contexts in the developing world. 

3. STUDENT IMPACT: Research on this question is likely to lead to 
substantial improvements in boys' and girls' learning and development outcomes 
in diverse contexts in the developing world 

4. RESEARCH FEASIBILITY: Research on this question is methodologically 
feasible in developing world contexts. 

The ranking of the 86 research questions was completed through an online survey. In 
this second survey, questions were presented by major thematic area (e.g., teacher 
development, school management and leadership). Respondents could opt out of 
sections if they did not have sufficient knowledge to respond. Thus, the number of 
respondents varies for different thematic areas because some surveys were only 
"partially" completed (the respondent completed at least one full section, but not all 
sections). The second survey was sent to 115 regional education experts including the 
original 84. 37 responses were received including 27 complete and 10 partial surveys. 
Examples of the questions and results are presented in tables 1 and 2. 
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Table 1. Example of Overall Research Questions and Results 


Theme 

Proposed Research 
Questions 

(n) 

System 

School 

Student 

Feasibility 

RPS 

AEA 

TL 

What pedagogical 

and classroom 

management 
strategies are 

effective in 

improving the 

quality of learning in 
large primary school 
classes? 

27 

0.85 

0.98 

0.95 

0.95 

93.17 

0.80 

ECD& 

G 

What types of ECD 
programs have the 
strongest impact on 
student learning over 
time? 

17 

0.94 

0.85 

0.94 

0.88 

90.44 

0.79 

SM&L 

What 

capacities/skills do 
primary school head 
teachers need to 
effectively perform 
leadership and 

administrative roles 
in different contexts? 

17 

0.88 

0.92 

0.89 

0.92 

90.33 

0.84 

TD 

What are the positive 
and negative factors 
affecting teacher 

attendance and 

absenteeism? How 
can these be best 
addressed? 

32 

0.88 

0.92 

0.89 

0.92 

90.23 

0.78 

ECD& 

G 

What kinds of 

training and 

credentials for early 
childhood program 
personnel best 

support effective 

pre-primary 
teaching? 

15 

0.93 

0.88 

0.84 

0.94 

89.74 

0.77 
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Table 2. Example of Research Questions and Results in a Thematic Domain 


TEACHING AND LEARNING 

N 

RPS (% ) 

What pedagogical and classroom management strategies are effective in 
improving the quality of learning in large primary school classes? 

27 

93.17 

What practices are most effective for student transitions from local to 
national languages and/or English as languages of instruction? 

26 

89.47 

What kinds of innovative non-traditional approaches to teaching and 
learning are happening in East African schools and classrooms and with 
what results? 

27 

89.45 

What kinds of supports are needed by primary teachers to implement 
multilingual education effectively? 

25 

88.92 

How are school systems and schools interpreting and responding to the 
demands for 21st Century Learning Skills in both policy and practice? 
What impact is this having on learning outcomes? 

26 

88.46 


The survey invited the respondents to identify their organizational affiliation and 
position. The characteristics of the 37 respondents to the second CHNRI survey were as 
follows: 

• Respondents: N = 37/101 

• Complete = 27 

• Partials (answered at least one section) = 10 

Geography (location of employment, not ethnic affiliation) 

• From the specific region of the developing world we focused on in this 

project (three countries represented) = 62% (23/37) 

• From the developed world (Australia, France, Switzerland, Canada, 

USA, UK) = 38% (14/37) 

Role (n = 35, 2 missing personal data) 

• Academic = 48% (17/ 35) 

• Non-Academic = 52% (18/35) 

Analysis of the CHNRI survey responses included computing the Research Priority 
Score and the Average Expert Agreement score (AEA) which shows the degree of 
agreement and variability across experts on the ratings for each question. While it is 
possible to weigh certain criteria based on the funding agencies, values, and focus 
(Tomlinson, et al., 2008), the agency which funded this project did not request any 
weighting. 
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Results of the CHNRI were presented by listing research questions both by overall 
ranking based on RPS score, as well as by thematic domain area where research 
questions were listed by their overall RPS rank under each theme. 

Rapid Reviews of the Literature 

In addition to the CHNRI exercise, our team completed a series of rapid reviews to 
synthesize research-based knowledge, issues and knowledge gaps identified in the 
literature. These reviews covered research literature internationally and regionally in 
five domains 2 (teaching and learning, teacher development, school management and 
leadership, parent and community and early childhood development and education). 

The rapid reviews utilized a strategic search method to identify key resources such as 
existing systematic reviews, literature reviews, reports and other gray literature from 
well-known and reliable sources on school improvement in general as well as in the 
contexts of developing countries and the specific region related to this project. To this 
end our team identified search terms for and recorded more than 765 individual 
searches (465 in Google and 362 in Google Scholar). We combed through and collected 
links to potential sources from approximately 4,135 pages of Google and Google Scholar 
search results. Completing the initial search, we identified 1) key sub-themes; 2) 
prominent authors and organizations; and 3) created a list of documents to be 
considered for the rapid review. We summarized key resources making note of 
significant findings, the evidence-base supporting these findings, and any knowledge 
gaps identified in the literature. 

The final documents included in the series of rapid reviews focused primarily on 
knowledge from extensive systematic reviews of the literature related to the sub-themes 
of this series, supplemented by recent empirical studies of particular relevance to our 
focus region and 10 other developing country contexts. The findings of this series of 
rapid reviews were reported alongside the results of the CHNRI exercise, and 
contributed to reinforce the overall findings in the prioritization exercise. 

Program Budgeting and, Marginal Analysis and Advocacy 

The results of the CHRNI process were provided in a detailed report to the project 
sponsor. This report helped to inform the sponsor organization in setting priorities for 
research and was then used as the basis for a call for research proposals. The OISE team 
was not involved in the actual selection of research projects that were ultimately funded. 

Reflections from the CHNRI Exercise 

As mentioned, we are not presenting the results of our CHNRI exercise in this paper. 
Instead, the following section highlights what we learned about applying the CHNRI 
methodology to education research in the context of the developing world. 

As with any research, there are limitations to consider that arise from the methodologies 
employed. It is important to remember that the CHNRI exercise results in a ranking of 

"Rapid reviews are available at the CIDEC website: 

http:/ /www.oise.utoronto.ca/cidec/Research/SESEA_Research_Consultancy.html 
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research topics based on the combined ratings of the experts surveyed using a set of 
criteria for judging the merits of those questions. The evaluation criteria used to rate the 
research questions on the survey were selected in consultation with the project sponsor. 
They are not the only criteria that could have been used, and perhaps the results may 
have varied had different or more criteria been employed. Some CHNRI studies use up 
to seven criteria, which we judged excessive for this survey. 

Greater response to the second survey (n= 37/115) may have provided more data; 
however, the current response rate of n=37 exceeds the acceptable levels for robust 
survey research. More importantly, reports on use of the CHNRI process in the health 
sector suggest that once 25-30 responses have been received, the rankings of research 
priorities do not substantially change as the "exercise reaches saturation" (Dean at al., 
2013, p.5). 

Our respondents came from a diverse group of academics and professionals working in 
international agencies, having expertise in education in the region of focus. However, 
few responses were received from the regional government education authorities we 
contacted. Furthermore, while a number of respondents (n=10) opted out from rating 
the questions of certain thematic sections, we feel confident that for the lower response 
sections the experts who chose to respond had sufficient knowledge in that particular 
thematic domain. However, in future CHNRI surveys, we would suggest targeted 
recruiting of experts in specific domains. 

One of the interesting findings from the CHRNI exercise was that the initial open-ended 
survey yielded a wide range of potential topics / questions for research but little 
consensus on those areas across the 22 experts who responded to that initial 
survey/interview. Only one or two experts explicitly named most topics. It was not 
possible to identify relative priorities amongst the different questions overall or within 
each major thematic area from this initial consultation. That fact confirmed the need to 
engage in a second stage in the research agenda setting process to enable prioritization 
of possible research questions. 

After completion of the analysis of the second survey we found the final rankings are 
relative; numerical differences between the questions are not great. The CHNRI 
procedure does not result in judgments of "significant differences" between items in 
statistical terms. Nonetheless, it does produce and reflect a sense of relative priorities 
drawing upon the knowledge of a broad range of experts in the field of inquiry. 

While the rapid reviews of the research in specific domains were used to supplement 
and triangulate the survey findings, the reviews proved especially useful in confirming 
the importance of the potential research priorities, and in highlighting potential research 
topics that were not captured in the survey process. 

Conclusions 

While the traditional CHNRI model is not directly applicable to education research, an 
adapted version of this methodology provides an evidence-based tool to help research 
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funders make tough choices as to what research should be prioritized. A research 
priority setting exercise of this nature is not able to provide definite answers on what 
research should be funded, but it does provide an excellent starting point for an 
informed discussion on what research should be prioritized. 

In addition to providing a detailed methodology for consensus building and priority 
setting, we found several other key benefits to the CHNRI method. As described earlier, 
our project involved participants from both the global north and the global south. This 
allowed us to incorporate the opinions and ideas of our colleagues in the global south in 
the research priority setting process, colleagues whose voices are not well represented in 
the literature. 

The creators of the CHNRI method believe that this priority setting exercise has the 
potential for broad application in the developing world (Rudan, 2012). Our experience 
with the CHNRI model supports this belief. Not only is this a systematic and 
transparent process for determining a research agenda, but it could also be applied to 
consensus development and priority setting in other areas of education investment as 
well. To this end, we believe that the CHNRI model has great potential for education 
decision makers in the developing world who may not have access to, or don't feel their 
perspectives are accurately or appropriately represented in the global knowledge base. 
The CHNRI method allows for decision makers in the global south to identify their own 
education experts, and provides a structured process that could support decision makers 
in the global south to set their own research agenda and in turn define their own 
priorities for development. 

In addition, this systematic approach for setting a research agenda provides the 
opportunity to bring academics, policy makers and practitioner experts, together who 
otherwise would not connect to each other, or would generally not share conversations 
to build consensus on a field of study. The education field may benefit from a greater 
number of interactions among experts which occur more regularly in other fields. 

To sum, an adapted version of the CHNRI method provides a practical, systematic and 
transparent approach to prioritizing education research investments on a local, national, 
regional or global scale. We hope that this paper provided useful information that can be 
built upon to support similar projects in education research investment. 

If you have any questions or comments on this piece, you may contact Robyn Read at 
Robyn.read@utoronto.ca. 
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